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ABSTRACT 



The DC Reads program was first implemented during the 



1997- 1998 school year as a partnership among the Corporation for National 
Service; District of Columbia Public Schools; Communities in Schools (CIS), a 
nonprofit corporation; and six local universities, with Georgetown University 
acting as lead. During the 1998-1999 school year, it expanded to include two 
more city-based universities and more nonprofit community organizations that 
coordinate the delivery of testing services. The Year 2 evaluation of DC 
Reads focused on the program operated in 16 schools by CIS during the 

1998- 1999 school year. Across its 16 sites (4 of which served as primary 
study sites) DC Reads provided tutoring to approximately 504 students during 
the school year- -most were in second grade. Students providing tutoring 
numbered 340; in the 4 study schools, 136 students received tutoring from 74 
tutors. DC Reads tutors were mostly Federal Work- Study students attending 
partner universities, and they used a curriculum and instructional materials 
purchased by the partners and adapted. Most schools used the Book Partners 
curriculum while a few schools used the Reading One-to-One curriculum. The 
Year 2 evaluation concentrated on assessing student outcomes . Findings 
include: significant gains on 5 of 8 tests in the Reading Performance 
Battery, with dramatic gains in phonemic awareness; lowest performers 
progressing at the fastest rate; and nearly twice as much gain for tutored 
students compared to nontutored students on the Stanford Achievement Test. 
Effective practices and lessons learned (solutions) are: use of 
research-based elements to produce reading achievement; well -structured 
tutoring sessions in which the content and delivery of instruction are 
carefully planned; close coordination with school personnel; intensive, 
ongoing training and supervision for tutors; frequent, regular tutoring 
sessions; careful evaluation, assessment, monitoring, and reinforcement of 
progress; access to training and technical assistance resources; and 
engendering positive, caring relationships among student, staff, and tutors. 
Students who received tutoring made significant gains in reading, and overall 
the program experienced considerable success in its second year. Appended 
are: methodology, sources of information, and outcomes. (NKA) 
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DC Reads Year 2 Evaluation 
Executive Summary 

A. Background 



The DC Reads program was first implemented during the 1997-1998 school year as a partnership 
between the Corporation for National Service (Corporation); District of Columbia Public 
Schools (DCPS); Communities in Schools (CIS), a nonprofit organization; and six local 
universities (American, Catholic, George Washington, Howard, and Trinity), with Georgetown 
University acting as lead. During the 1998-1999 school year, the program expanded to include 
two more city-based universities, Southeastern and the University of the District of Columbia. It 
also expanded to include more nonprofit community organizations that coordinate the delivery of 
tutoring services. 



This Year 2 evaluation of DC Reads focused on the programs operated in 16 schools by CIS 
during the 1998-1999 school year. Four of the schools served as primary study sites for the 
evaluation. 



Across its 16 sites, CIS DC Reads provided tutoring to approximately 504 students during the 
1998-1999 school year. Most tutored students were in second grade. A total of 340 tutors 
provided tutoring to those students. In the four study schools, 136 students received tutoring 
from 74 tutors. 



DC Reads tutors, most of whom were Federal Work-Study students attending partner 
universities, provided tutoring to first- through third-grade students twice a week up to an hour 
each time. Tutors used a curriculum and instructional materials that the partners purchased and 
adapted. Students who participated in DC Reads were those identified by school staff as having 
low reading skills and being most in need of tutoring. 



At the beginning of the 1998-1999 school year, most schools used the Book Partners curriculum 
while a few schools used the Reading One-to-One curriculum. At the start of the second 
semester, a new blended curriculum incorporating both was adopted in all schools. The blended 
curriculum enabled tutors to meet the needs of a wide range of students who received tutoring, 
including those with very low reading skills. 
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B. Outcomes 



The Year 2 evaluation concentrated on assessing student outcomes, although it also included a 
program implementation assessment and an investigation of program achievements. 

1. Significant Gains on 5 of 8 Tests in the Reading Performance Battery, With 
Dramatic Gains in Phonemic Awareness 

The purpose of the outcome assessment was to determine whether tutored students’ reading 
skills improved after participating in the DC Reads tutoring program. The central element of the 
outcome study was the Reading Performance Battery, which consisted of eight individually- 
administered reading assessments. Most tests in the battery were administered three times. 
Between the fall and spring testing (Wave 1 and Wave 3), tutored students demonstrated 
statistically significant improvement on five of the eight measures in the Reading Performance 
Battery. Students improved in letter identification, phonemic awareness, word attack, and oral 
reading fluency and comprehension. Their most dramatic gains were in phonemic awareness, 
the matching of sounds to their letter symbols. This is considered to be a fundamental skill, 
prerequisite to the development of more advanced reading skills. 

There were no statistically significant declines in scores, even on the subtests that are adjusted 
for time on a monthly basis. On six of the eight tests, the standard deviation of scores decreased, 
a change indicating that students performed not only better but also more consistently as a group 
in Wave 3 testing. This means that fewer students were falling far behind. 

2. Lowest Performers Progressing at the Fastest Rate 

By using Hierarchical Linear Modeling (HLM), a statistical analysis technique, the evaluators 
determined there was significant variation among tutored students in their initial scores on 
measures of phonemic awareness, word identification, and passage comprehension. For each of 
these outcome measures, students who had lower scores tended to grow at a faster rate than those 
with higher scores. This indicates that the lowest-performing students were “catching up” in 
some areas. HLM also indicated that students who practiced reading aloud more frequently 
during tutoring improved at a higher rate than students who did so less frequently. 

3. Nearly Twice as Much Gain for Tutored Students Compared to Nontutored 
Students on the Stanford Achievement Test 

Tutored students made greater gains on the Stanford Achievement Test (SAT 9) reading tests 
than did nontutored students. On Normal Curve Equivalent scores (which compare students to 
all other students in the nation at the beginning and end of the school year), tutored students 
gained nearly twice as much as nontutored students. Also, 69 percent of tutored students 
improved on Total Reading, while just 60 percent of their nontutored peers demonstrated 
improvement. 
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Although tutored students performed significantly below nontutored students in both fall and 
spring testing, their SAT 9 score improvements offer evidence that tutored students made 
significant gains above and beyond what would be attributable to the reading instruction received 
in their classroom. 

C. Effective Practices and Lessons Learned 

The Corporation defines effective practices as the activities and methods a program engages in to 
produce positive results. Lessons learned are the solutions implemented in the program or 
envisioned as improvements for future programs, often developed in response to the challenges 
the program confronted. This study uses the Corporation’s previously identified 1 eight principles 
of high-quality tutoring programs to present the effective practices and lessons learned from the 
Year 2 DC Reads program. 

Macro collected data related to program implementation and achievements by means of 
interviews, observations of tutoring sessions, various program recordkeeping logs, and survey 
questionnaires. 

1. Use of Research-based Elements to Produce Reading Achievement 

The CIS DC Reads program used Book Partners and Reading One-to-One, two well-tested 
reading curricula developed at major universities to improve the reading abilities of low- 
performing students. (Book Partners is a DC Reads adaptation of Book Buddies, which was 
developed at the University of Virginia; Reading One-to-One was developed at the University of 
Texas by George Farkas.) The selection and use of these curricula comprised the effective 
practice component of the program while the adaptation of the model (e.g., blended curriculum) 
to better meet the needs of students reflected a lesson learned. A second lesson learned was that 
the blended curriculum still did not meet the needs of all children in the program, particularly 
those who had severe reading difficulties and those who progressed but did not progress enough 
to be graduated from the program. 

2. Well-structured Tutoring Sessions in Which the Content and Delivery of 
Instruction Are Carefully Planned 

The CIS DC Reads tutors used formal written lesson plans to guide each tutoring session. The 
curriculum materials were geared to a wide range of reading abilities, and there was a 
corresponding lesson plan for each book. One lesson learned was that the large manual 
developed to guide the lessons in one curriculum was unwieldy and was not well received by 
tutors. The second lesson learned was that the increased range of materials and clear lesson 
plans relieved tutors of having to invent their own activities during tutoring sessions. 
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The principles are described in the Corporation’s Principles and Key Components for High-Quality America 
Reads and National Service Program Initiatives and are grounded in the literature on reading research. 



3. Close Coordination With Schools, School Administration, and the Classroom 
and/or Reading Teacher 

Related to close coordination, the CIS DC Reads program involved teachers in selecting the 
students for the tutoring. Second, the main DCPS office was involved in the program and 
encouraged principals to take an active role in supporting the program. The presence of 
AmeriCorps*VISTA members as program coordinators at each participating school provided a 
point person for school staff. Providing information about the program to school staff at the 
beginning of the school year was also an important effective practice. 

Improving access to facilities and equipment, increasing communication with teachers, 
increasing access to the school-based reading specialists, and recognizing how cultural 
(organization and ethnicity/race) differences impact communication comprise the lessons learned 
related to close coordination. 

4. Intensive and Ongoing Training and Supervision For Tutors 

For this program element DC Reads had two effective practices. One effective practice was 
hiring an individual with a background in reading to coordinate all DC Reads training activities. 
Another effective practice was to have designated individuals, in this instance 
AmeriCorps*VISTA members, onsite in each participating school to supervise the tutors. The 
lesson learned was that relying on existing staff within DCPS to provide training to DC Reads 
tutors and others did not allow the program to deliver the training required in a timely fashion. 

5. Frequent and Regular Tutoring Sessions 

Establishing a specific time and days for the tutoring sessions was an effective practice in CIS’ 
DC Reads program. The lessons learned related to the actual number of tutoring sessions that 
were held, as it was lower than expected. The actual length of each tutoring session was also less 
than the specified amount of time and the length of sessions varied from school to school, with 
some schools scheduling 40-minute and others 60-minute sessions. 

6. Careful Evaluation, Assessment, Monitoring, and Reinforcement of Progress 

The effective practices related to evaluation, assessment, monitoring, and reinforcement of 
progress were establishing uniform procedures and forms to document student progress and 
using student SAT 9 scores as a basis for teacher referral to the program. Another effective 
practice was making needed program modifications at mid-year — implementing the blended 
curriculum. One lesson learned related to the overall monitoring of tutors and program quality as 
this was not yet performed to the extent program managers believe is necessary. A second lesson 
learned was related to the program evaluation. An important outcome evaluation finding was 
that the group-administered SAT 9 tests provided useful information about student progress. 
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7. Access to Training and Technical Assistance Resources 

In the CIS DC Reads program, one effective practice related to training and technical assistance 
is within-school access to a reading specialist. Another is access to the program’s trainer/reading 
specialist for additional support. The lesson learned was that while in theory a school-based 
reading specialist was available to the program in most schools, the reading specialist was not 
actually involved in many schools. 

8. Engendering Positive, Caring Relationships Among Students, Staff, and 
Tutors 

Tutors worked to develop positive relationships with the students they tutored. School staff and 
others believed that there were a number of academic and other benefits to the children from 
having the opportunity to interact one-on-one with a caring young adult. The use of Federal 
Work-Study students as tutors was an effective practice. The lessons learned were that tutor 
recruitment was difficult and that tutor turnover did not allow established relationships to 
flourish. 

D. Conclusion 

Students who received tutoring made significant gains in reading, and overall the program 
experienced considerable success in its second year. The experiences of the partners 
implementing the DC Reads program offer valuable lessons to others involved in planning or 
implementing similar programs. In Year 3, the partners expect to continue to build on previous 
successes while addressing the remaining challenges, primarily in the areas of tutor recruitment 
and retention and communication among partners. 



Chapter I. Student Outcomes 



A. Outcome Study Overview 



The Year 2 evaluation of DC Reads focused on programs operated in 16 targeted 
assistance schools by Communities in Schools (CIS). Four of the schools served as 
primary study sites for the evaluation. 

Students who participated in DC Reads were those identified by school staff as having 
low reading skills and being most in need of tutoring. 

The Year 2 evaluation concentrated on assessing student outcomes 1 , although it also 
included a program implementation assessment and an investigation of program 
achievements. 

The purpose of the outcome assessment was to determine whether tutored students’ 
reading skills improved after participating in the DC Reads tutoring program. The central 
element of the outcome assessment was the Reading Performance Battery. 



Macro International Inc. conducted the Year 2 evaluation of DC Reads for the Corporation for 
National Service. The evaluation focused on programs operated by Communities in Schools 
(CIS). CIS operates DC Reads programs in 16 schools designated by DCPS in 1997 as the 
schools in most need of academic assistance. Four of the 16 schools served as the primary study 
sites for the evaluation. CIS was also the focus of the Macro’s Year 1 evaluation of DC Reads. 

Tutored students were identified by principals or teachers as those being in the greatest need of 
assistance in reading. Identification was based upon students’ previous year Stanford 
Achievement Test (SAT 9) scores, below average reading grades in the previous year, poor in- 
class reading performance, home environment, and other factors. 

In each of the four study schools, Macro randomly sampled 20 tutored students to participate in 
the outcome study. On the basis of fall 1998 SAT 9 scores, tutored students who participated in 
the outcome study did not differ significantly in reading achievement from those who did not. 
However, students selected to receive tutoring did have substantially lower reading skills than 
nontutored students. 



Measures of student outcomes included individually administered tests in a Reading Performance Battery and the 
Stanford Achievement Test (SAT 9). The tests in the Reading Performance Battery were administered by 
qualified educational testing specialists specifically for the evaluation. The SAT 9 tests are routinely administered 
to all students in the DC Public Schools in the fall and spring of each school year. Other data related to student 
outcomes included an assessment of students’ attitudes toward reading, students’ reading grades, and school 
attendance records. 



B. Data Sources for the Outcome Study 



1. Reading Performance Battery 

The battery of tests used to assess reading performance consisted of eight standardized 
assessments. The tests were individually administered by professional educators to students 
at the four schools selected as primary study sites. Most of the tests were administered three 
times during the school year: Wave 1 in October and November 1998 (n=85), Wave 2 in 
January and February 1999 (n=75), and Wave 3 in May 1999 (n=51). The timing of testing 
and tutoring was as follows: 



Wave 1 testing — Begin tutoring — Wave 2 testing — End of tutoring — Wave 3 testing 



Figure 1: Reading Performance Battery Tests 2 
Identification of Capital Letters 

This test measures whether the student can recognize the 26 capital letters. Students are asked to identify 
randomly ordered capital letters. Scores are expressed as the percentage correct. 

Identification of Lower Case Letters 

This test measures whether the student can recognize the 26 lower case letters. Students are asked to identify 
randomly ordered lower case letters. Scores are expressed as the percentage correct. 

Yopp-Singer Test of Phoneme Segmentation 

This test provides a tool for assessing children’s phonemic awareness and identifying children who may experience 
difficulty in reading and spelling. The test measures a child’s ability to articulate the sounds of a spoken word 
separately and in order. For example, given the orally presented word “sat,” the child should respond with three 
separate sounds: /s/-/a/-/t/. (The sounds, not the letter names, are the appropriate response.) There are 22 items in 
the test. Administration of the test normally requires about 5 to 10 minutes per child. Scores are expressed as the 
percentage correct. 

Woodcock Reading Mastery Test Subtests (Forms G and H) 

Various types of scores are available for all subtests of the Woodcock Reading Mastery Test, including standard 
scores and grade equivalent scores. For most analyses in this study, standard scores were used. The following 
subtests were administered: 

Word Identification. The Word Identification subtest requires the student to read isolated words. The items 
range in difficulty from words typically presented in beginning reading programs to words difficult even for 
above-average college students. In Form G, there are 106 words on the list, beginning with very common 
monosyllabic words and ending with infrequently used multisyllabic words. 



2 Most tests in the Reading Performance Battery were administered three times during the school year. The 
exceptions are the Oral Reading Time and Comprehension subtests of the Durrell Analysis of Reading Difficulty. 
This test, available in only one form, was administered only in Wave 1 and Wave 3. 
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Word Attack. The Word Attack subtest measures students’ ability to apply phonic and structural analysis 
skills in pronouncing words not recognizable by sight. The test includes both nonsense words and words that 
are used very infrequently. The word list begins with words that require only basic decoding skills and 
progresses to words that are much more difficult to decode. 

Passage Comprehension. The Passage Comprehension subtest measures the student’s ability to study a short 
passage — usually two or three sentences long — and to identify a key word missing from the passage. About 
one-third of the Passage Comprehension items are one sentence long and contain a picture related to the text. 
Forms G and H of Passage Comprehension each contain 68 items, arranged in order of difficulty. 

Durrell Analysis of Reading Difficulty Subtests 

Two subtests of the Durrell were administered for the study. 

Oral Reading Time 

For this test, students read prose aloud and then answer corresponding comprehension questions. The test 
presents eight passages in order of increasing difficulty. Students continue to read passages and answer 
questions until they make seven or more errors, or until the time required for reading the paragraph is outside 
the time norm. The instructional reading level is defined as the median grade on the time norm. The levels 
are, Primer, 1L, 1M, 1H, 2L, 2M, 2H, 3L, 3M, 3H, and so on. (In each case L stands for Low, M for Medium, 
and H for High, so that 3L indicates a low third-grade level). The scoring is determined by taking into 
account factors such as reading rate and number of word errors. 

Oral Reading Comprehension 

This test uses the same passages as Oral Reading Time. Students are assigned a score of “good,” “fair,” or 
“poor.” If there are no more than two errors in answering the questions, the student’s comprehension is 
considered “good.” If three questions are unanswered or answered incorrectly, the student’s comprehension is 
considered “fair.” More than three questions unanswered or answered incorrectly results in a rating of “poor.” 



2. Stanford Achievement Test Scores and Language Arts Grades 

We obtained the Stanford Achievement Test (SAT 9) reading scores, as well as the language 
arts grades and attendance for all second graders in the four study schools. The SAT 9 is 
used by DC Public Schools (DCPS) to assess its overall performance. A primary reason for 
obtaining the SAT 9 scores was to determine whether there was a strong correlation between 
the SAT 9 subtests and individually administered tests. 3 If so, this would have considerable 
cost implications for future evaluations of DC Reads and other tutoring programs. We also 
obtained students’ grades in language arts in order to ascertain whether there is a strong 
relationship between students’ reading performance (as measured by the Reading 
Performance Battery) and their grades. 4 



3 Tesls selected for the battery were individually administered to ensure that the results would not be affected by 
how well a child could follow directions in a group setting. A strong correlation between the SAT 9 and the 
battery would indicate that students had sufficient test taking skills and that both sets of tests measure the same 
constructs. 

4 In addition to students’ grades, the attendance records of all second grade students were obtained. The school 
attendance levels of students who were tutored were not significantly different from the school attendance levels 
of their generally higher performing nontutored classmates. We also assessed tutored students’ attitudes toward 
reading, but found no statistically significant change from program start to program end. There was also no 
systematic relationship between students’ attitudes toward reading and their reading performance. 
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